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ABSTRACT 

This feasibility study contains a design for 
evaluating vocational education research and de\/^elopment (R&D). The 
study developed a model for evaluating vocational education R&D 
based on legislative frames of reference for research, exemplary 
projects, and curriculum development. Called an Impact Tracer Model, 
the program is based on information and observations taken from 
distribution records of 308 vocational education R&D products, 
utilization data from 12 exemplary products distributed nationwide, 
and impact data from 6 field sites. The five-stage model consists of 
needs assessment , product development, product dissemination, product 
use, and product impact. Some of the conclusions reached through the 
study are the following: (1) distribution data are easier to acquire 
and less expensive to obtain than product use or impact data; (2) 
most research coordinating unit directors are willing and able to 
provide distribution data; (3) superficial data on use can be 
obtained by mail, but detailed documentation requires observation or 
interviews; (4) baseline measurements should be established when 
assessing impact; (5) interactive methods are necessary for 
collecting impact data; and (6) data on product use and impact are 
collected by few states at present. The study recommends that a 
compreheasive and continuing vocational education R&D evaluation 
system be implemented in phases beginning with the collection of 
distribution data on all qualified R&D products. The appendices 
include survey instruments and a conceptual framework for assessing 
impact. (KC) 
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FOR H: WORD 



Vocational education research and development (R^D) 
represent-R one of the prirnary means of improving practice in 
educational agenv.,ies at the local, state, and national levels. 
RSiD can he effective as a ..program improvement strategy only if 
government officials make prudent investments of scarce re- 
soMirces. This technical report helps them do this by proposing a 
system for evaluating the effectiveness of vocational education 
R&D. It contains an evaluation model, stra teg ies "for imple- 
menting product assessments/ and options for operating a 
comprehensive evaluation system at different levels of effor;,.- 

This report is intended for policymakers and decision makers 
in the federal government and in state governments who allocate 
resources for research and development (Rf»D) in vocational 
education. These are n,S, Office of Vocational and Adult 
Fducation staff, research coordinating unit directors, and, 
curriculum coordinating center directors, among others. 

The information in this report should be used to build more 
accountability into RfiD activities at each level of government* 
Evaluation activities must be designed f.or the unique needs of 
each state and/or project. The outcome of a comprehensive 
evaluation system should be increased R&D impact ori classrooms 
and students. 

v;o acknov/ledge the assistance of David ?• - Crandal 1 , 
Executive Secretary of The Network, Inc., and Doren Madey^ 
Pinnell, Director of Corporate Relations, The Fuga School of 
Business, Duke University, who. reviewed draft manuscripts of this 
report. Comments from M. Catherine Ashmore, Program Director of 
Product Management for the , National Center and Wesley E, Budke, 
Program Director for the National Center Clearinghouse, on the 
draft manuscript also were appreciated. Assistance in typing the 
manuscript was provided by Jeani Gray and ed i tor i a 1 ass i s ta nc e 
was provided by Conn ie " f add is . 
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FXRCFJTIVR SUMMARY 



This feasihility study report contains a design for 
evaluating vocational education research and development (RStD). 
Very few components of this design are currently "in place" 
nationwide. These components -^re intended to help build a 
continuing and comprehensive evaluation system of vocational 
education RffD. 

This study developed a model for evaluating vocational 
education research and development, based on legislative frames 
of reference for research, exemplary projects, and curriculum 
development. These RffD activities include Programs of National 
Significance and state-administered projects. The model suggests 
a relatively low-cost evaluation system that uses existing 
reporting mechanisms at the national, state, and local levels. 

The model is based on information and observations taken 
from (1) distribution records from 308 vocational education R&D 
products, (2) utilization data from twelve exemplary products 
distributed nationwide, and (3) impact data from 'six field sites. 
These data are presented in a companion document entitled 
Summary of Dissemination Outcomes (hull 1983). Additional 
insight was obtained from state-of-the-art literature reviews, 
suggestions from consultants, and experiences with prior impact 
assessment activities. 

The five-stage Impact Tracer Model describe J in this report 
assumes the following: 

~ o Program improvement is issue-oriented and site- 
specific; that is, the worth of a product for re- 
solving a problem on-site depends upon its relevance 
to the prohlem(s) at that site. 

o The primary output from a funded R&D project , 

normally a R&D product, is the most usable vehicle 
for tracing effects of the projects. 

o A high-quality R&D product, by itself, will not cause 
program improvement to occur; most cases require 
involvement of user audiences. 

o Acceptance and use of an innovation developed 

elsewhere is less ex^^onsive to an adopting unit than 
developing the innovation. 

o Accountability is the driving force behind most 
impact studies, although some reference to impact 
studies as baseline measures for future longitudinal 
studies may be found in the discussion of the model. 
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The five stages are (1) needs assessment, (2) product 
development, (3) product dissemination, (4) product use, and 
(5) product impact. Measurement criteria are presented in each 
stage addressing each of the following questions respectively: 

o What was the problem being addressed? 

o Did the product meet or exceed development 
standards? 

o Where di(j the product go? 

o What happened to the product at its 
destination? 

o What difference did the -product make? 

At the development stage, the criteria are presented as quality 
control standards. The first stage should be viewed as an 
attempt to identify the problem being addressed by a particular 
vocational education R&D project. The second stage is the 
development of a solution for the problem, in the form of a 
product. Stages three and four comprise the delivery system for 
transporting and implementing the results of R&D at particular 
sites. Stage five assesses the impact of the R&D product 
following an appropriate amount of lapsed time since introducing 
the product. Conclusions drawn in this report suggest the 
following: 

o Distribution data is easier to access and less 

expensive to obtain than product use or impact data. 

o Most research coordinating unit (RCd) directors are 
will ing and able to provide distribution data, 
although some states are not organized to provide it 
by standard categories. 

o Superficial data on ut il izat ion can be obtained by 
nail, but detailed documentation requires observation 
and/or interviews with users. 

o Basel i ne measurements should be es tabl ished when 
assessing impact. 

o Interactive methods (e.g., telephone and face-to-face 
interviews) are necessary for collecting impact data 
under mr>st conditions. 

o Da t a on product use and impact are col lec ted by very 
few states at the present time. 
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A comprehensive and continuing vocational education R&D 
evaluation system is recommended. It should be implemented in 
phases beginning with the collection of distribution data on all 
qualified R&D products. Product use studies should b*- targeted 
to specific samples of product recipients to determine if result 
are being applied to resolve problems. Impact studies should be 
conducted on high-priority needs only. Baseline Llata should be 
collected in advance of product use whenever possible. 
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CHAPTER 1 
INTRODUCTION 

This report describes options for evaluating vocational 
education research and development (R&D). Vocational education 
R&D is a diffuse enterprise, bounded somewhat by time and by 
levels of funding from federal and state governments. The report 
discusses the need for a continuing and comprehensive evaluation 
system and describes different segments of the enterprise. This 
chapter also includes a statement of objectives and a description 
.of the methods used to conduct the feasibility study. Chapter 2 
presents the evaluation model,' including detailed descriptions of 
components. Chapter 3 discusses the options for implementing the 
model system. The final chapter summarizes the feasibility study 
and recommends an implementation option. 

Need for Comprehensive Evaluation of 
Vocational Education R&D 

r 

The Vocational Education Act of 1963 ( F . r,, • o8-21 0 ) 
contained the first broadly-based provisions for vocational 
education Rf,D in recent history. It authorized funds for 
research, training, and experimental pilot programs. The 
Vocational Education Amendments of 1968 (P.L. 90-576) revise.d the 
funding categories to include research and training, exemplary 
programs (demonstrations), and curriculum development. The 
Education Amendments of 1976 ( P. L. • 94-4 82 ) adjusted categorical 
funding by segmenting program improvement (Subpart 3) into 
research (Section 131 ) , exemplary and innovative programs 
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(Section 132), and curriculum development (Section 133). 
Personnel training (Section 135) and vocational guidance and 
coun<=;elinq (Section 134) also were listed under Subpart 3. 

House Resolution 4164, proposed as the Voca t iona 1--Techn ica 1 
Act of 1983, contains authorization for research programs and 
curriculum development not only under basic grants to states 
(Title II), but also within program improvement (Section 305). 
Career guidance and counseling programs and vocational education 
personnel training programs also are listed under Section 305. 

Throughout these acts, which authorize hundreds ot millions 
of dollars, program improvement themes of research, demon-^ 
stration, and personnel and curriculum developr^en t have occurred 
consistently. For example, a National Center for Research in 
Vocational Rducation was authorized under these acts and has been 
funded since 1965, These relatively homogeneous functions within 
program improvement have been ongoing for nearly two decades. A 
clear and present need exists for a system to evaluate their 
effectiveness. 

A number of major evaluation studies have triced to determine 
the impact of vocationa] education R&D (e.g.. Development 
Associates 1975; General Accounting Office 1974; Hu and 
Stromsdorfer 1975), but clear or consistent findings have been 
scant. Magisos and Moore (1977), after reviewing these 
evaluation studies, suggested that only programmatic efforts 
could adequately collect, process, and analyze data necessary for 
assessment of vocational education PfrD. More recently, Kim 



2 

12 



(1^82) reviewed three nationwide studies and four state-level 
studies of RfiP impact. His suininary revealed positive effects of 
Rf«n f)rojects in vocational education, hut noted several 
limitations in interpreting the results. Chief among the limi- 
tations was the absence of direct observations and statistical 
significance tests of R&P project impact. There were few 
indications of "how much" impact occurred as a result of the R&D. 
There v;ere conceptual and methodological problems with the 
studies. 

Another reviewer of nationwide impact studies, Mohamed 
(19R3)r leveled three criticisms at the studies overall: 
(1) they lacked research design, (2) sample sizes were too small, 
and (3) descriptive studies were overabundant. The Committee on 
Vocational Education Research and Development (COVERD) (1976) had 
difficulty determining R&D impact, because measures often have 
been "subjective and difficult to validate and have failed to 
measure the long-term effects of R&D" (p. 3). Limited resources 
and constrained time lines have led the director of the 
Vocational Education Study (David i9R3) to plead for "systematic 
fol]ow-up on the consequences of federal policy ... as the data 
become available" (p. 13). A system for evaluating the impact of 
vocational education R&D needs to be established. 

Some steps have been taken to build more accountability into 

vocational education R&D. The current authorizing legislation 

ft 

for vocational education, Public Law 94-482, contains the 
following expectation for vocational education research: 

3 . 

13 



No contract shall he made. . .unless the - appl i can t can 
demonstrate a reasonably probahility that the contract 
will result in impro\7ed teaching techniqv^'^s or curriculum* 
"materials that . will be used in a substantial number of 
classrooms or other learning situations within five ye^ars 
after the termination date of such contract. (Section 131 b) 

As recently as 't1ay 1983, a question was added to the U.S. 
Department of Education's form ED 590 requiring project abstracts 
to contain descri.pt ions of processes used to assure high-quality 
products^ (Office of Vocational and Adult Education 1983). 
Information on the number of products produced from government- 
sponsored projects, their u-se, and their perceived impact could 
be. collected from product users. For example, Haney (1980) of 
the Huron Institute argues for targeted evaluation research aimed 
at answering specific questions. Such a strategy, he says, 
"could be easily coordinated with a federal approach to 
evaluation of vocational education" (p. 4). 

This feasibility study examines options for collecting data 
through a comprehensive eva uation system. The recommendations 
in chapter 4 may serve as a corners tone i n the framework for 
evaluating the impact of vocational education R&D* 

Objectives of the Feasibility Study 

The objectives of this feasibility study are as follows: 

o To develop a model system for evaluating vocational 
edu ca t ion research and de ve lopme'n t 

o To describe the costs and benefits of evaluative 
information provided by this system 

o To determine the feasibility of implementing the 
system at different resource levels 



The first-, objective is addressed in chapter 2? the second and 
third are discussed in chapter 3. A strategy for implementing 
the model is recommended in chapter 4. 

The, purpose of data collected through an evaluation system 
is to document the distribution^ use, . and impact of vocational 
education Rf^H products. This information is to be used for 
accountability and planning purposes. Primary users are 
government officials at the federal, state, and local levels, 
persons who are more interested in accountability than planning 
because they must accept responsibility for spending large 
amounts of taxpayers' money. Other audiences include those who 
are greatly interested in data collection methods, as well as 
results in program improvement processes. They are vocational 
education stakeholders, the Research Coordinating Unit (RCU) 
directors, the Curriculum Coordinating Center (CCC) directors, 
and others instrumental in funding, conducting, and reporting 
proje ct resul ts . 

Methodology Used to Develop This Design 
A four-pronged approach was used to establish the viability 
of this evaluation design. First, a technical review of project 
objectives and methodology was conducted by National Center staff 
members early in 1983. Underlying assumptions were explored and 
expectations established for the mode] evaluation system. 
Second , pro je ct abs t ract s and other li tera ture on impact 
evaluation from computer searches of EPIC and other document 
databases were reviewed and a ^conceptual framework was developed 
for the eval\jation system. 
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This conceptual framework guided the development of the model 
evaluation system. Third, this framework was shared with RCU 
directors at their annual meeting in March 1983, and with the 
National Network for Curriculum Coordination in Vocat ional and ^ 
Technical Education (NNCCVTE) at their directors' council meeting 
in April 1983. The resulting framework and both deliverables of 
this project were subsequently reviewed by outside consultants. 
Fourth, observations were collected via three modes , telephone, 
from of f -the- she 1 f records , and from in terv lews at six field 
sites, to establish the availability and feasibility of acquiring 
evaluation data. 

This final thrust was used to refine questions for assessing 
the usefulness of vocational education RfitD. Appendix A contains 
the evaluative questions submitted to the Office of Vocational 
and Adult Education (OVAE) for instrument clearance. These 
questions were based on results obtained from dissemination and 
utilization studies completed during the previous five-yef^r 
contract with the U.S. Office of Vocational and Adult Education 
(Hull 1980, Hull 1982). Impact questions were developed from a 
broad base of R&D product studies, including the following: 
Adams, McKay, and Patton (1Q'81); Anderson and Hull (1981); Bragg, 
Hull, and Adams (1981); and Modisette and Bonnett (1981). Impact 
criteria from these studies were summarized in a report by Hull, 
Adams, and Bragg (1983). 

Limitations of This Research 

It became necessary, early in this project, to limit the 

scope of this inquiry because of the diversity of projects and 
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'quantity of products funde.i through vocational education program 
improvement,^ Project staff decided to apply a fairly strict 
definition of research and development associated with program 
improvement in vocational education* This interpretation adheres 
to the logic of the leg isla tion and includes only those projects 
funded in research, exemplary/innovation, and curriculum 
development categories. Personnel training and vocational 
guidance and counseling projects are excluded. 

A second limitation is related to R&D project output^. Not 
all projects funded with federal monies resulted in products,* 
Even when they did, not all products were made available to the 
Educational Resources Information Center (ERIC) Clearinghouse on 
Adult, Career, and Vocational * Education • Clearinghouse staff 
estimate that approximately halT the programs of national 
signif icance and/or. state-administered projects result in 
documents accepted for inclusion in the ERIC database. 

Despite these limiv.a tions , data collected for this 
feasibility study came from a wide variety of states (twenty-two 
of the thirty-two states with products in the ERIC database at' 
the time the sample was selected). Communication with primary 
users (e.g., teachers and instrument users, such as RCU 
directors), was direct and informative. This occurred despite 
wide differences in the way states maintain their records and 



*P'ederal funds flow to project directors through one of two 
sources. Either the projects are Programs of National 
Significance or they are state-administered program improve- 
ment pr6j ec :s f unded f rom federal mon ies rece ived as part of a 
'formula grant, 
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gr^^at. variability in tbp rolps anci organizations of the persons 



contacted. In general, the techniques used to collect data, were 
relatively efficient. 

The evaluation rnodel described in the following chapter is 
intended as a model system capable of being used at the federal, 
state, or local level. A state or district may wish to implement 
a part or all of it. The criteria in the model may be applied- 
superficial ly (e.g., a content analysis of project abstracts) or 
in-depth (e.g., in a series of interviews during site visits). 
The i ntens i ty of the da t a collect ion activities wil 1 depend upon 
the level of perceived need for the data and the resources (i;e., 
time and money) available to collect it. 

Assumptions of the Model 

The following assumptions und^rgird this evaluation model: 
o Program improvement is issue^or iented and site- 



specific; tba t is, the wortb of a product for 
resolving a problem on site depends upon its 
relevance to the problem(s) oi;, that site. 

o The primary output from a funded RffD project, 

norma 1 ly a RfiD product , is the most usable vehicle 
for tracing effects of the project. 

o A h igh-quali ty R&P product, by itself, will not cause 
program improvement to occur; most cases require 
i nvol vemen tofuseraudiences. 

o Acceptance and use of an innovation developed 

elsewhere is less expensive to a,n adopting unit than 
developing the innovation. 

o The model gives priori ty to account abil ity as the 
driving force behind most impact studies; although 
some reference to impact studies as baseline measures 
for future longitudinal studies may be found in the 
discussion of the model. 
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The f^ystem design for the evaluation model i.q to be 
continuing and comprehensive; that is, the design should include 
all aspects of. U^D activities^ particularly including moasuroinent 
of the impact of products. An emphasis is placed on products as 
outputs of R&D projects f because products represent a tangible 
vehicle for transporting innovative ideas. No attempt has been 
made to downplay or set aside the many innovative and develop- 
mental contributions of local teachers and administrators. In 
factr they represent the primary audience for many R&D products 
because they have to make the innovations work in their 
particular setting. However, in this evaluation model the 
emphasis is on development of products that can overcome the 
necessity of "reinventing the wheel" in educational settings. 
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CHAPTER 2 
THE EVALUATION MODEL 

At the. heart of this system for evaluating vocational 
oducation RfiD is an evaluation model capable of being imple- 
mented at the federal^ state, or local level. This chapter: 
(1) explains the model developed for evaluating vocational 
education R&D, (2) describes components in detail, and (3) 
discusses strategies for collecting data. 

The substantive focus of this evaluation model is program 
improvement resulting from the use of R&D products. It includes 
both Programs of National Significance sponsored by the U.S. 
Office of Vocational and Adult Education and projects authorized 
through Sections 131, 132, and 133 of P.L. 94^482. The focus 
includes projects of research, exemplary progr*ams, and curriculum 
development. Sections 134 and 135 of P.L. 94-482, personnel 
training and vocational guidance and counseling, are not included 
in the scope of this project. Projects authorized by sections 
134 and 135 most often clo not involve research and development 
activities. 

(^ther models have included a wider scope of activities.. The 
Florida Research Coordinating Unit model (1982), for example, 
included vocational guidance and personnel development within a 
program improvement framework. McCage (19B0) included impact 
analysis following each of four components in a program 
improvement continuum. Researchers at the University of Illinois 
(Evans et al . 1982) examined three dimensions when designing 
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impact assessments: short term-long term^ intended-unintended 
consequences, and direct-indirect effects. Hughes (19B3) 
recommended a production framework for research within an 
input-output framework, where output included products, effects, 
and impact. 



Background for Model Development 

As indicated in the previous chapter, the vocational 

education R&D enterprise is widespread and diverse. It is 

defined uere as including Programs of National Significance and 

state projects funded by program improvement federal monies . It 

includes applied research, curriculum development, exemplary 

programs, and on occasion, personnel development. To encompass 

all of these efforts, it was necessary to select an evaluation 

model capable of spanning a wide range of activities and of 

focusing on the audiences instrumental to program improvement ir 

vocational education. 

Several approaches to assessing the effects of R&D were 

considered; among them were the following: 

o Exemplary site case studies . Th is approach to 
assessment looks for "best case" examples of R&D 
product use, then investigates them to determine what 
impact has been made. Case studies help us to 
understand why a product works, but the general- 
izability of results often are quite limited. The 
representativeness of the site may be difficult to 
establish, and costs of collecting interview data are 
high. Th i s approach to impac t as5^essment can be 
useful in establishing a "high water mark" measure of 
impact, but it should be used in conjunction with 
other assessment methods . 

o Random sample of practice situations . Th is approach 
to assessment randomly selects classrooms , state 
agencies, or other groups of vocational educators who 

12 
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r^ight. logically adopt R&D products. Organizations 
are stratified in advance, hut thex.^ are very few 
limitations placed on the selection of practice 
units. Personnel in these units (e.g., teachers, 
principals, and so forth) are interviev;ed to deter- 
mine if they have used any'P&P products. This places 
a burden on them to remember the product and identify 
it with a particular practice occurring in the 
classroom. Often, people prefer to claim personal 
responsibility for a promising practice, and they do 
not remember where it came from. This approach was 
rejected as inefficient and ineffective. 

<^ R&n product tracer studies . This approach to assess- 
ment focuses on specific R&D products with the intent 
of describing their dissemination, use, and impact. 
One or more products may be studied simultaneously on 
.one or more subject areas. A limiting feature of 
"this approach is its exclusive focus on products? 
effects associated wi th part ic ipat ion in group 
activities or other nonproduct-rela ted activities 
may therefore be lost. This strategy can be cost- 
effective if records are kept to support the 
objectives of the evaluation. 

The "product tracer" approach to impact studies was chosen 
for the' evaluation model because it fits many of the assumptions 
made in the conceptual framework. It is relatively, low-cost in 
its data collection at the early stages of the model and it 
provides flexibility in the amount and quality of data produced. 

The model illustrated in figure 1 should be read from left 
to right and from top to bottom. It contains five distinct 
activities: (1) needs assessment, (2) product development, 
(3) product dissemination, (4) product use, and ^(5) product 
impact. These impact ' evaluation elements correspond closely to 
the categories in the conceptual framework . Appendix B contains 
an explanation of the conceptual categories in the evaluation 
f ramework . 
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Needs Assessment 



Specifications for R&D Impact: 



Determine the problem (discrepancy) to be addressed 
List the number and type of persons to be served 



Identify expected changes In Indi vidua 
behavior or organizational arrangement 



^ 

Product Development Standards for 
Qua I ity Control ; 

• Attacks a problem at a point amenable • Meets a perceived need as judged • Meets 80 percent of development 

to change by 90 percent of field trial criteria at a good to excellent 

users performance level during field tr I 



JZIZ 

Strategies for Data Col lection: 

Product Distribution 

Formative Dissemination 
Measurement Criteria: 

• Strategic dissemination 

• Multiple dissemination 

• ;Wldespread dissemination 



Figure U Traoar model of R&D Impact evaluation 



Product Us'9 




Product Impact 


Formative LTtlllzatlon 
MeasuremePit Criteria; 


> 


Summat 1 ve E f feet s 
Measurement Criteria: 


• 1 ntegrated ut 1 1 I zat Ion 




• Increased productivity 


• With support systems 




• Cost savings 


e With t Ime on task 




• User satisfaction 
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Needs Assessment 

The. needs assessment component of the model emphasizes the 
importance of problem identification as the first step in 
evaluation of vocational education R&D. This component 
represents a systematic attempt to identify problems amenable to 
resolution^ through research. The success of an R&D project 
should be measured by its attainment of standards established 
diirinq the needs assessment process. This process is easier to 
describe than it is to implement. 

In reality, any identifiable problem is affected by a 
multiplicity of factors. Use of an R&D product is only one of 
many. The process of docuren t i ng need is fraught with 
difficulty. As Adams suggests (1983), needs sensing often fails 
to provide insightful and specific information. Fdijcational 
goals are so general that it is difficult to obtain precise 
information. Also, controversial needs often are buried among 
measures of centra.'' tendency, which cluster ratings into a narrow 
middle range. 

These technical problems are not discussed here. Instead, 
the reader is referred to the following documents, which review 
needs assessment processes and suggest measurement instruments/ 
techniques. For example, the National Center produced a paper 
(Ahmann I'^Vfi) on needs assessment for program planning in 
vocational education. It defines needs assessment as a 
discrepancy between agt'ual and desired conditions, it reviews 
nepds assessment methods, and discusses systems models for 
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planning vocational education programs based :)n empirical data. 
Another National Center publication (McCaslin and Lave ]97f;) on 
needs assessment focuses on car-ier education. It describes 
available needs assessiTt?nt instruments in this area and discusses 
approaches for statewide assessme n ts . V7i tk i n ( 19 7 5) provi des an 
excellent review of needs assessment technigues for educational 
planning at the local, intermediate, and state levels. The 
instruments reviewed tend to focus on substantive areas (e.g., 
career education, multicultural education, reading, etc.). For 
additional examples of needs assessment instruments, see the 
Educational Needs Assessment Handbook produced by the Arizona 
Department of Fducation (1976). 



Product Development Standards for Quality Control 

Fxamples of guality control standards that should be met 

prior to release of an PStD product are as follows: 

o R&D product(s) should contain clear objectives , 

indicating not only the components of the product, 
but the extent of product use, as well. 

o Fach product should contain a detail ed product cost 
analysis, including not only the anticipated 
operating costs, but installation and modification 
costs, as well. 

o Information supporting cla ims for the product should 
be compelling and reported in sufficient detail to 
allow judgments on its merit or worth to potential 
users. 

o Fvidence of product effectiveness should rule out 
rival hypotheses, such as maturation or practice 
effects, and should establish causality between the 
product and program changes. 

o The product should be free of bias related to 

raci a 1/e thn i c group S5i^,^ sex, special needs popula t ions , 
and occupational s terdtyp inrj . 
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o The product should be easy to^se , readily divisible 
. into operational programs, or adaptable for 
innovative activities.' - 

The standards for quality control shown in the impact 

evaluation model (see figure 1) are at a different level of 

specificity than those just mentioned. The product development 

standards in figure 1 are more general and contain operational 

language typical of field trial formative evaluations;. Standard 

must be simple, nonambiguou s , acceptable to persons developing 

the products, easy to use, attainable within the resource 

limitations of the project, and acceptable to persons using the 

product s. 

development of an RfiD product should follow a systematic 
process. There are many authoritative references on this 
process. Raker and Schutz (1971) have produced an informative 
volume on instructional development. Pratzner and Walker (1972) 
have edited a similar volume on programmatic R&D in vocational 
education. 

Strategies for Data Collect ion 
There are few limits on strategies for collecting impact 
data. Fvaluators should allow their imagination and .the 
a va i la bi 1 i ty of i n forma t ion , unobt ru s ive and otherwise, to 
dictate assessment approaches. This section briefly describes 
three strategies and f ive tech n iques for da ta collect ion . 

Figure 2 lists the strategies and areas for eva lua-t ing 
product impact'. The matrix suggests salient techniques for 
collecting data related to three of the five questions included 
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Data Collection 
Strategies 


Product 


Distributive 


Use 


Impact 


Product 
Census 


• Records 

• Mail surveys 






Product 

Sample 

Surveys 


;|§ 


• Telephone 
interviews 

• Mail 
surveys 


ill 


Case 
Studies 






• Face-to-Face 
interviews 

• Baseline 
measurements 



Figure 2. Illustrative data collection strategies 
for evaluating product distribution, 
use f and impact 
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in the tracer model of R&D impact evaluation. The ciuestions are 
as follows: 

o Where did the product go? (distribution) 

o 'what happened to the product at its destination? 
( use ) 

o What difference did the product make? (impact) 
The distribution, use, and impact stages of the model are defined 
in the Glossary of Terms (appendix C). Answers to each of these 
questions suggest different approaches to assessment. 

Product Dissemination 

The first question of what happened to the product(s) 
suggests a record keeping approach resulting .in a census of all 
product copies distributed. The rationale for this recom- 
mendation is simple:^ Project directors should be accountable for 
expenditures of federal funds. Auditors should be able to track 
the number of copies produced at the public's expense. These 
records provide a trail leading to persons likely to be prime 
users of t he' produc t ( s ) . A census is also valuable in assessing 
the proportion of the primary audience reached through dis- 
tribution of product copies This statistic is a key in 
determining whether the problem identified through the needs 
assessment is being addressed. It may be necessary to use mail 
surveys to determine whether relevant audiences have received 
copies of the product. 

Product Use 

'The second quest ion about product use 1 j kewi se is relevan t 
to evaluation of R&D impact. Credible evidenc oL' product use 
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must be available to establish causal linkages between the 
product and changes occurring in vocational education. At the 
minimuFi, these linkages must be inferred* 

A survey of product recipients, either by mail or telephone, 
may be used to determine how the R&D product has. been used. This 
is a major commitment of resources; however, it may be necessary 
to. establish the fidelity of product implementation. Data on 
estimates of cost feasibility, frequency of use, and the degree 
to which the pro^duct has been integrated throughout the 
organization often provide insights into how the product(s) have 
been used. 

Product Impact 

The third question about product effects addresses complex 
issues of change and program improvement. Baseline data are 
needed in order to establish changes in a program before and 
after product use. Rarely, is this kind of information available, 
and comparison groups often cannot be found to serve as a basis 
for e.'^timating program change. This is why the case study is 
'•relied upon as a data collection strategy for assessing impact. 
.Intervening variables can be considered as they influence 
site-specific conditions. 

Of course, case studies represent a narrow focus for impact 
assessment because they are expensive. A very limited number of 
studies can be done to gain an understanding of the deeper, more 
involved issues associated with R&D-based program change. These 
data supplement results from survey questionnaires and rating 
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forms. Face-to-face interviews that follow up initial non- 
directive questions with specific probes based on previous 
responses often yield worthwhile qualitative data. Such studies 
require highly trained interviewers who are sensitive to subtle 
cues. 

More needs to be said about the feasibility of using 
selected techniques for assessing impact. Five major data 
collection techniques are discussed next. 

A record keeping approach may be most appropriate for 
superficial levels of data collection. Most R&D organizations 
disseminate outputs (products) from projects to particular user/ 
audiences. At least one state has such a listing of an audience\ 
(e.g., industrial arts teachers ) in a computer data bank? it is 
possible to print current address labels of these teachers with 
the press of a button. Almost any state department of education 
could adopt a "quick and easy" method of recording copies of 
products disseminated to select audiences. First, however, 
fundamental information, such as role and organizational affil- 
iations of recipients, should be agreed upon. The categories 
contained in questions submitted to OVAE in appendix A may be 
considered a step in this direction . 

Evaluators need to know more, however, than just the numbers 
oZ product recipients in order to judge the potential impact of 
R&D products . They also need to know the degree of product use 
by the recipient • This calls for some type of interactive 
communication between the evaluator and the product recipient. 



At the very least, a mail survey should be used to determine how 
much use has taken place. 

A mail survey , as a technique for assessing impact, can be 
effective for quantitative answers to predetermined questions. 
It allows quick and easy tabulation of results associated with 
standardized variables. A mail survey is least effective when 
variables need to be clarified or more information is needed to 
understand the response given. Often, impact data'^ require some 
knowledge of the context or conditions influencing product use 
on-site. Often use is not clear cut, or the product has been 
modified by the recipient to fit the conditions of the school or 
classroom. The questions in the Product Use Survey (see appendix 
A) include definitions of potential user categories that may be 
used to arrive at uniform data. Such questions may also be used 
with telephone interviews. 

A telephone interview adds a new dimension to impact 
assessments. It allows the evaluator to follow up leads 
contained in responses by asking related but different questions. 
These can be very important when determining impact, because 
often an in terviewee does not ' recogn i ze some activities as 
product use. For example, sharing a publication with other 
teachers may be considered use by some teachers but not by 
others . The telephone interview affords an opportun ity for the 
eval ua tor to probe for deeper , more mean ingf ul responses to 
assessment questions . 

VJhen telephone interviews are used, it is important for them 
to be scheduled at a time convenient to the interviewee. This 
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may hp extremely difficult to do, and inappropriat'c=? timing of an 
interview is a major disadvantage of th-is data collection 
<=;trategy. Administrators may he caught at husy times, or 
teachers may not have had time to think about their use of 
materials during the field test telephone interviews. Scheduling 
telephone interviews usually requires several advance calls, hut 
it is worth the effort if the interviewee is prepared for the 
interview. 

Face -to- face interviews , as a data collection technique, 
allow the interviewer to observe nonverbal reactions to the 
questions. These can be quite helpful in identifying areas for 
further pursuit of impact. Most often, face-to-face interviews 
take place on the interviewee's home territory, allowing 
observation of environmental factors that may influence the use 
and effects of P&D products. Site visits are expensive, but they 
represent one of the most complete ways to rneasure impact. 

Basel j.ne measurements are needed to assess long-range 
i:Tipact. This suggests planning and commitment of resources to 
document existing conditions in advance of introducing the R&D 
product. Large-scale projects may be able to afford special 
assessments to establish baseline measurements for the project. 
Most RfcD efforts in vocational education will have to rely on 
needs assessment data used to identify the problems addressed by 
the PFtD project. Measures taken to identify the problem (e.g., a 

c 

statewide survey of parents' views on the quality of vocational 
education) may or may not be representative of conditions at 
local site(s) where the P^D product has been introduced. 
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Committing funds for multiyear longitudinal studies also presents 
a major barrier for most project directors trying to establish 
baselines. 



Criteria for Measuring Impact 

Regardless of the strategies or techniques used to collect 
data, certain criteria must be met to measure impact The tracer 
model of R&D impact assumes it is possible to assess impact by 
looj^dng.at preliminary indicators. Thus, the model includes 
impact criteria at the distribution, use, and effects stages. 

Impact criteria from the conceptual f ramework 's ,di s- 
semination and use stages are presented ne5ct. The effects stage 
criteria, which are clearly different from the other two stages, 
are discussed in greater detail^ 

Dissernination Criteria 

Strategic dissemination . Cost-ef f ect ive strategies 
for disseminating an R&D product should be devised 
based on characteristics of potential users, 
site -specific factors, and features of the product 
i tself . Dissemi nation should be stra tegi c in 
reaching opinion leaders and influential organi- 
zations in the external environment. 

Multiple channels . More than one channel for 
conveying information about innovations should be 
used. Communication should include mass media (e.g., 
direct mailing of brochures) and interpersonal 
^channels (e.g., technical assistance). Norma 1 ly , , 
i nf ormat ion duplication and overlap are assets rather 
than liabilities during the dissemination stage . 

Widespread d isser^i nat i on . In nova t ions should reach 
as many potential users as poss ible. Thus, dis- 
semi nation to individuals in different roles, in 
diverse settings, and in many geograph ic areas should 
be emphasized. Secondary dissemination through 
workshops, reprint s, 1 ibraries, the FRIC system, and 
so on should be encouraged. 
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utilization Criteria 



Integrated use . The use of an innovation product 
should be in tens ive and pervas ive throughout the 
organization . Accomplishing this task will require 
drawing on personal commi tment within the organ i- 
zation to institutionalize the product and fit it 
into organizational routines. 

Support systems . Support systems necessary for 
encouraging the full use of an innovation should be' 
operational at the time of implementation. These 
systems are of three types: personal resources 
(e.g., administrative endorsement, site personnel 
endorsement), information resources (e.g., training 
in the use of support materials and procedures), and 
physical resources (e.g., dollars, supplies, and 
equipmen t ) . 

Time on task . An R&D product should be used 
f requen tly enough and long enough for its use to 
become an integral part of current practice. The 
audience's time in actually using the product should 
maximized. ^ 



Effects Criteria 

Effects criteria are summative, as compared to formative, 
and some require elapsed time before they can be measured.* 
Unlike the two previous stages, two of the three criteria in the 
effects stage deviate from criteria identified in the conceptual 
framework. They are related to cost-benefits derived from R&D. 
Policymakers' increased emphasis on productivity and cost savings 
is. noteworthy because of trends limiting resources for vocational 
education R&D. 

Increased productivity , as a criterion for measuring the 
effects of vocational education R&D, can be applied to either 



*This is elapsed time between the introduction of an R&D 
product into a setting and the measurement . of changes in that 
setting resulting from use of the product. 
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individuals or orga n iza t ionR . it takes the form of increased 
efficiency, most often in organizations (e.g., the sane task can 
be performed quicker or with fewer persons). Increased effic- 
iency is so common an element in field data that a separate 
criterion of cost savings is included in the effects category. 
This criterion of cost savings shows up as time saved in 
preparing lessons by teachers. Curriculum materials and resource 
guides are used most often because they save teachers time in the 
library or in preparing paperwork for class. Occas iona 1 ly , 
increased productivity appears as a genuine increase in output, 
such as an increase in students' test scores. It also takes the 
form of improved papers completed for class assignments (e.g., 
more footnotes and references used in the papers). 

Measuring R^iO impact as outputs of the educational process 
rather than as improved efficiency of inputs should be a major 
goal of most evaluation plans. Many barriers to long-range 
measurement of outputs exist? some, such as longitudinal funding 
and lack of appropriate experimental field-test design, have 
already been mentioned. In addition, it becomes very difficult 
to establish causal linkages between the R&D product use and any 
increases in productivity measured three or five years later. 
Many events intervene between the introduction of an R/stD product 
and the long-term measurement of its effects. For these reasons, 
case studies are recommended as the best strategy for measuring 
J^FiT) effects; /\ 



User sat isf act ion , th/e third criterion in the effects stage, 
addresses an affective dimension of impact measurement? that is. 
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did fhe person like the product? This variahle, more than the 
other two, can be manipulated by persons implementing the 
product. Satisfaction is a function of meeting expectations. 
Administrators introducing a product have an opportunity to 
influence expectations either positively or negatively; the level 
of expectation for product performance developed by product 
recipients influences their degree of personal satisfaction. 
User satisfaction may be indicated by. product advocacy and/or 
creative adaptations in use of the product. 

Criteria for measuring R&D impact must be uniquely 
applicable to the project being studied. Criteria should be 
substantive and site-specific. R&D program evaluators should 
expect to select criteria to fit the purpose(s) of the evaluation 
and to use data collection strategies consistent with the 
resource ]imitations of the evaluation study. 
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CHAPTFR 3 

STRATFGIRP FOP IMPLRMFNTTNO THF SYSTEM 

The degree of implementation associated with a "continuing 
and comprehensive" evaluation system of vocational education R&D 
depends upon answers to several key questions. Among the 
questions are the following: 

o Wil? the data collected be significant? Will the 

information obtained from the evaluation system be 

used to influence key decisions, such as appro- 
priations for R&D projects? 

o Can the desired information be obtained? Any data 
collection wi 1 1 be a compromise between what is 
possible (given the amount of resources allocated) 
and what is desired. Will the information obtained 
be sufficiently compelling to justify the resources 
expended? 

o Can the cost be contained within reasonable limits? 
What is reasonable for one evaluation inquiry may not 
be reasonable for another. Relatively limited data 
are needed to support an audit to determine whether 
program improvement funds were spent for their 
intended purpose. Seeking data to support or reject 
claims of high-quality P&D outputs, for example, 
will require a much greater expenditure of funds. 

The need for a continuing and comprehensive evaluation 
system is clear. A better system for documenting the effects of 
R&n activities is possible. Missing is t he recomme nda t ion for a 
specific evaluation system. 

This chapter contains several implementation strategies 
discussed within the context of three important questions: 

(1) should data collection be comprehensive or selective; 

(2) what is the least-cost alternative for collecting required 
evaluative information; and (3) do local, state, or national 
agencies have the respons ib i 1 i ty f or collecting impact data? 
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Comprehensive versus Selective Data Collection 
Iir.plf^men t i ng a continuing and comprehensive evaluation 
system for vocational education Pf*D does not mean collecting 
information for all data categories — distribution, use, and 
effects — or for all products. In fact, the cost of such a system 
would be prohibitive. Least-cost alternatives associated with 
the collection of distribution data may use the electronic 
mailbox or the conventional mail system. Additional costs will 
be minimal if information collecting and reporting requirements 

can be incorporated into^ existing systems. For example, dis-^ 

tributi'D data could be included with abstracts of products 
delivered to the National Center Clearinghouse. 

Although it may be feasible for a pro je ch or state to 
collect distribution data on most products, rarely — if ever-- 
would uSv? data be . collected on all products. Cosbs associated 
with finding the recipients, asking them questions about product 
use, and analyzing the results would be prohibitive for all 
products. Selected studies of products designed to resolve high- 
priority problems, however, would be feasible. Product recipients 
could be stratified by some demographic variable important to the 
evaluation study, such as role and organization. Random samples 
of recipients could then be taken to determine whether key 
audiences have not only received the product, but have used it as 
well. The type of use, as indicated in the questions in appendix 
A , rou 1 d [>rovi do r hios to t ho likely resolu t i on of the prohli-^m . 
These selected studies of product use might be commissioned by 
some government agency on an "as needed" basis. 

30 

39 



The collection of impact data to measure effects of product 
use needs to be very selective. Establishing baseline 
information as a benchmark for judging the effectiveness of R&D 
in resolving problems requires advance planning; in most cases, 
it will involve multiyear studies and clear concepts of the goals 
to be achieved. Unforeseen factors often intervene to delay or 
diminish the effectiveness of longitudinal evaluations. These 
factors range from staff changes to cuts in funding. The need 
for selective information about priority concerns should be the 
guiding influence on the design and conduct of impact studies. 

Alternative Cost Scenarios 
The following scenarios were constructed as options for 
implementing and operating a continuing and comprehensive 
evaluation system under conditions of scarce resources. Each 
scenario has a dollar figure attached to it. Cost estimates are 
difficult to formulate for several reasons. 

First, the purpose and use of the evaluation results dictate 
the conditions of tho data collection, such as the degree of 
confidence to be placed in the data, the size of the sample, and 
so forth. Sometimes the questions relate to use of products; at 
other times the investigations revolve around the impact of 
products • 

Sf^cond, the type of products and the actual products 
selected for study included in the assessments affect the 
selection of respondent audiences and the nature of the dependent 
measures themselves. For example, a study of information paper 
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use could easily focus on university classrooms or the number of 
times the information papers have been quoted by professors. A 
curricu lum product , on the other hand, would be evaluated by 
teacher acceptax;ice and on its ability to increase students' 
capabilities. Any given study of R&P impact will be influenced 
greatly by the particular products "chosen for study. A' random 
selection of products produced in a particular theme area could 
identify different types of products, unless the products were 
stratified by type. The specific products identified, with their 
respective lists of product recipients, would determine the size 
of the population. It could be in the hundreds, or much less. 

Third, the method(s) used to collect the data will be 
determined by the respondent population, sample size, and the 
amount of resources (i.e», time and money) available to conduct 
the study. Telephone interviews, for example, take a lot more 
time than mail questionnaires. Trade-offs in the quality and 
quantity of data are very real. 

These three factors--the purpose of the evaluation study, 
the actual products selected*, and the methods used to collect 
evaluation dat a--i n teract to complicate the formulation of cost 
est^imates. respite these limitations in estimating costs, 
several options are proposed in this report for comprehensive 
evaluation of v6catior\al education R&D. The cost estimates 
associated with^each option are based on the following 
assumptions: 

» . 

o Evaluative information about product distribution, 
use and impact is sequential. Fach stage precedes 
the other, so information about product distribution 
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is necessary to understand product use. Information 
about product use is necessary to understand product 
impact. 

o The sampling units and the units of measurement are 
different for each of the stages. Product dis- . 
tribution samples » are based on individual products, 
but the measurement units are in number of persons 
receiving the product. Product use , on the other 
hand, is determined by the different kinds of uses, 
often quantified by the number of persons involved in 
that type of use (e.g., members in a class, or the 
•frequency of use by a person). Product impact 
-measures or effects result from actions 9ccurring in 
the previous two stages (distribution and use). 
Impact focuses on issues, such as whether the problem 
has been resolved.' The sampling units. for impact are 
likely to be people and organizations in a position 
to resolve the problem. The measurement units, on 
.. the other hand, may be fragments of the problem being 
resolved. For example, the problem of school drop- 
outs has many dimensions, but one amenable to 
resolution through vocational education is the abilty 
of students to acquire employ ab i 1 i ty skills. 

fDocumenting resolution of the problem may take the 
form of (1) increased student test scores, (2) in- 
creased placement rates, and/or (3) improved average 
daily attendance figures. The burden of proof for 
showing problem resolution — or progress towards 
it; — still rests with the'^project director -and the 
funding agency. The logic of the needs assessment 
should, be used to identify areas of the problem 
. vulnerable to attack. Evaluation data should be 
provided accordingly. The exact sampling scheme and 
units of measure are dependent upon the strategy for 
using R&D products and the people, places, and things 
affected by the use of these products. 

o The number of 'products evaluated is inversely related 
to the sequenced stages of distr ibut ion , use , and 
ef f ec ts . For any g iven problem to be resolved , 
distribution data will relate to a greater number of 
/ products than use data. Likewise, the expense of 
collecting effects (impact) data will limit the 
number of products considered to a lesser number than 
products evaluated at the use stage. 

Within each of these data collection stages — distribution, 

and ef f ec ts--there are certain minimums to be observed to 



ensure the collection of reliable and accurate data. This 
question of accuracy is addressed in the Request for Clearance 
docun^ent submitted to the sponsor as part of the year one ( 1983) 
scope of work. The number of products and the number of 
respondents at each data collection stage will depend on the 
level of accuracy desired' by the sponsor. 

The distribution level proposed in table 1 is a census 
eliminating the need for a sample. At the use level, it is 
possible to use relatively small numbers if random selection of 
both products and product recipients can be achieved. For 
example, as few as seventy--two respondents are needed per product 
to ensure accuracy in product use studies under the conditions of 
random selection of products and random selection of product 
recipients.' Persons making claims for product use under these 
conditions can be 95 percent confident that the data are at least 
90 percent accurate. This level of detail should be left to the 
design of the specific evaluation study. 

A range of costs is provided in table 1 for each option as 
an aid to planning. The estimated number of products needed to 
provide credible information for each option is based on the 
prior experience of the authors. Each scenario describes 
evaluation activities at the project, state, and national level. 
Each refers to Programs of National Significance and state- 
administered program improvement projects. 

The scenarios address dissemination/diffusion activities of 
R^;n products. Needs assessment and product development 
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TABLE 1 



OPTIONS FOR COLLECTING R&D PRODUCT IMPACT DATA 



el of Effort 
ions for Imple- 
ting an R&D 
luation System 


Cumulative 
Cost Estimate 


Types of Data Collection 


(Midrange) 


Distribution 


Use 


Effects 


Product Distri- 
bution Census 


$49/300 


■1 

Products : 540 

Respondents : 60 

Cost: $33,050 - 
$65,550 


■ 




Selected Use 
Studies 

o 4 Products/ 
Problem 

o 75 Respon- 
_ dents/ 
Product 


$120,350 


Same as above 


Problems: 2 

Products: 8 . 

Respondents : 600 

Cost: $66,300 - 
$75,800 




Investigative 
Impact 

o 10 Sites/ 
Product 

o 35 Respon- 
dents/ Site 


$191,100 


Same as above 


Same as above 


Problems: 1 

Products: 1 

Respondents: 

Cost: $63,75 
$77,750 



^Annual cost estimates to the fede^Tal government do xiot include indirect costs. 
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evaluations ar^ omitted because these activities should be 
.completed prior to (1) project initiation, or (2) product 
release.* 

The cost estimates of each option are based on the ^following 
rates: 



Personal ( $6 5 , OOO/st udy unit) 

o Research spec ial ist 

o Gradua te research associates 

o Clerical support 

Duplication 

o Questionnaires and Reports 

Postage 

Servi ces (telephone) 

Travel 

o Meetings 

o Rite visits 



Budget Rates 

$35,nOO/FTE 
15,n00/FTK 
15,000/FTR 



.OS/page 

.05/ 

quest ionna i re 

4no/study 



7nn/trip 
2n0/site/ 
person 



These cost estimates do not include payment to respondents for 
completing the questionnaires or interviews. Also, they do not 
include indirect costs to the sponsor. The cumulative costs 
estimates in table 1 subsume the previous options; so the cum- 
ulative estimate for option 3, for example., includes activities 
covered in options 1 and 2. Persons interested in an 



*Omission of these activities does not mean they are less 
important; rather it suggests that needs assessment and product 
development evaluations will need to be completed prior to impact 
assessment in most cases. 
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.mpact study of two ..products , for example, would multiply the 
rost estimates times two. 

Tho cost (Estimates a.s.sociatod with tho options in table 1 
ire cumulative because of the need to know where the products 
/ere distributed before the degree of use can be estimated 
lationwide. Likewise, t he ex t erVt ' of any changes discovered 
ihrough effects studies can be generalized only with knowledge of 
ise patterns. The three stages are linked together for compre- 
lensive and continuing impact assessment. One segment may be 
.mplemented in isolation from the others, but a corresponding 
^oid in knowledge of the other stages would result. 

Scenario No. 1; Product Distribution Audit 

The objective of this study is to document the number of 
iignificant R&D products produced and distributed to primary user 
udiences. Proigraij improvement products are those produced from 
esearc , exemplary, or development dontracts funded in part or 
n whole with reder 1 dollars. During a recent five-year period 
FY 1978-19G2) there were nearly 3,994 projects funded at a total 
evel of $104,638,* ^ (Budke 1983). This amounts to approxi- 
lately prr -^r- per year. Assuming one product per project 

nd a 60 percent response rate, data would be available on 
pproximately 480 state-administered projects. 

Records obtained from RCU directors will indicate the number 
f copies distributed at the time the product was first produced, 
t will not be possible to determine how many microfiche have 
een copied from ERIC files or how many times the product was 
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shared with other professionals. However/ the task of collecting 
this initial data is relatively simple, RCU directors in the 
states are responsible for their program improvement activities. 
Experience suggests that they are willing to provide distribution 
data on products produced from f ederal ly funded projects . The 
average RCU director would invest approximately two and a half 
hours per year providing these data. In some states r products 
are distributed by project directors; RCU directors could acquire 
these records from local project directors in these states and 
transmit them to a national location, where the records would be 
summarized. '* • 

Programs of National Significance would be summarized on the 
same type of record form as the ones used for state-administered 
projects, A listing of the projects and their directors is 
available from the Projects in Progress file at the National 
Center Clearinghouse, Project directors would mail their 
distribution records to either the National Center or the U,S,^ 
Office of Vocational and Adult Education, Sixty products came 
from these programs in 1983, 

At least two potential ways exist for reporting product 
distribution statisticjy from :individual projects. The 
recommended methpd is to attach a couple of questions about the 
role and organization of recipients to the product abstract 
submitted to the National Center Clearinghouse, This would 
eliminate the need for a new reporting mechanism. The infor- 
mation would be summarized by the National Center 'Clearinghouse 
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staff. The alternative method for reporting the product 
distribution data would be through the accountability reports 
completed by each state department of education. These data 
would be summarized and reported by staff in the Office of 
Vocational and Adult Education. The unit of measurement would be 
in number of copies distributed. (See appendix A for the 
specific distribution measurement questions.) 

Scenario No. 2; Selected Use Studies 

The objective of these studies is to determine R&D product 
use. The wide variety of products ranging from information 
papers to research reports makes the selection of products to 
study a critical decision in the conduct of use studies. The 
example used in this option examines utilization of eight 
products. This immediately suggests a case study approach, with 
quant i tat ive ^ da ta collected from surveys of product users. In 
this example, telephone surveys are used to compute the upper 
cost estimate, and mailed questionnaires are used for the low 
es t ima te . ' 

The eight products could be very different, such as an 
information paper, a teacher curriculum guide, and a program 



procedures guide for administrators. Alternatively, the products 
could be of the same -type (e.g., eight different research 
reports). The selection of products and the degree of inter- 
relatedness will affect costs. In general, a certain amount of 
synergism exists when products related to the same problem are 
studied. For this reason, the general recommendation is to pick 
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products carefully, preferably ones that are likely to reveal 
insights into the resolution of a common high-priority problem. 

The '600 telephone interviews in this scenario would draw 
from the questions in appendix A. Numbers of product users are 
important because they give a sense of quantitative use. 
Likewise, the length of time used and the amount of depth 
indicate qualitative use. Impressions of use are gained from the 
frequency of responses to questions. The types of audiences most 
affected by the use (e.g., classroom students) can be established 
from the interviews. Also, a sense of product acceptance or 
rejection is communicated . read ily . 

The researcher is faced with a difficult task in reporting 
this information. How does one communicate impressions? Both 
qualitative indicators and quantitative data can be used to 
support logical but tentative conclus ions . from these 
observations. Quotations can be used to portray representative 
beliefs and opinions. Tables are appropriate for comparing 
number of users, for example, in different types of settings. 

Reports of ► these selected usf? studies a^'e likely to be 
highly focused on one or two important questions, such as whether 
products are being used by teachers to improve instruct ion 
provided to vocational students. The results convey what ts 
happening in a community, state, or nation, but not what 
difference the products make. Some indications of what 
respondents think might happen as a result of use can be 
obtained. This information is speculative unless the survey is 
done some time after the product was introduced. 



The5=?e surveys also provide an opportunity to gain impres- 
sions about the quality of iihe product. Results from users can 
be combined with evidence from product developers to arrive at a 
considered judgment of quality. This is also an opportune time 
to obtain feedback on dissemination methods and to solicit 
suggestions for improving the delivery of R&D products. This 
kind of information does not by itself indicate product use, but 
the suggestions for. improvement r if implemented, could add to the 
cumulative impact of future R&D products disseminated. 

Selected product use studies could be conducted by a variety 
of agencies, including the National Center, third-party 
evaluatorsr or staff at the U.S. Office of Vocational and Adult 
Rducat ion . 

Scenario No. 3; Invest igative Impact 

The third option indicated in table 1 is labeled' 
"investigative impact." Even with the establishment of an 
evaluation system for collection of data, impact will always be 
elusive. The objective of this study is to determine whether R&D 
product use makes any difference'. To a degree, every impact 
study is an open-ended investigation. Leads are uncovered 
enroute to documenting changes. These ought to be pursued if 
(1) they are relevant to the study and (2) time and resources 
permit. 

The impac t opt ion in the table 1 matrix examines the use of 
one product in each of ten sites. On-site interviews of product 
users and others are suggested for the following reasons: 
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o Important 'nopverbal cues can be obtained in 
face-to-face interviews. . 

o Factors influencing impact, either positively or 
negativelyr can be observed. 

o Economics of data collection can be obtained by 

interviewing several people at the same site. This 
also helps to gain alternative views of what 
happened. 

o More than one person can be interviewed at the same 
t ime r thus allowing confirmation or rejection of 
information provided. . 

o • More than one interviewer can be taken to a si te r 
thus allowing comparisons of notes and interactive 
planning. 

o It is easier to establish baselines for a site than 
for a group of sites, or for an entire state . Many 
details "wash out" in group comparison data. 

Reasons for interviewing people in their home environment vary 

but, in general, first-hand knowledge of situations helps to 

analyze and interpret impact measures. 

Measures of product effects wil.l be quite different for 

different kinds of products. For example, the impact of an' 

information paper may be measured by the number of times the 

paper is cited in subsequent research or in policy decisions. 

Indicators of curriculum changes, on the other hand, may be found 

in the growth of students' knowledge in the substantive area of 

the materials.' The substance of the program improvement is very 

important in the measurement of impact. Growth in a student's 

ability to learn to weld, for example , would be measured qui te 

differently than growth in a student's ability to cook. 

Indicators of impact should be established for each substantive 
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area. Look for individual growth, organizational change, and 
effects on the community. 

This cannot be done without documenting the individual's or 
organization's performance level at the time the R&D product was 
introduced. Achievement tests^ records or organisational 
productivity, and so forth are very important as baseline 
measurements of change. In the absence of such records,, 
investigators must piece together fragments of evidence c» 
indicating how the^ individual or organization "used to functio.n." 
This can be done by asking questions about changes that have 
taken place, or by describing conditions prior to the 
introduction of the product. But this information is less 
satisfactory than an independent measure of performance at a 
prior point in time. Of course, the ideal situation — and one 
that ought to be followed in matters where large sums of 
resources have been invested — is to construct the baseline 
measurement in advance of introducing the product. Rarely does 
this happen, but the frequency is increasing as agencies become 
more interested in obtaining information on R&D product impact. 

This information would be reported in much, the same manner 
as the product use information. Special reports would be 
provided to the sponsor by the contractor. The contractor could 
be an independent agency, the National Center, or the U.S. Office 
of vocational and Adult Education. 
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utilization of Impact Evaluat ion Results 
Once data have been collected, what should be done with the 

results of an impact assessment? How will the evaluation make a 

difference in R&D programming? 

At the needs assessment stage, conduct of an evaluation can 

help ensure- rational, data-based decision making. It's difficult 

r 

to prioritize' needs under the best of conditions. A formal needs 
assessment can" systema t ical ly collect information on diverse 
needs. 'This procedure should help administrators think through 
logical consequences of funding decisions, including who should 
receive research products and what changes can be expected if 
they use them. 

Evaluation of R&D products as they are being developed is 
essential for high quality products. Product ^use data' from field 
test sites can reveal worthwhile information on such variables as 
teacher acceptance of th.e innovation, student growth, and ease of 
\ise. Reviews of products for biases, such as occupational 
stereotyping, can prevent inadvertent disiseminat ion of 
undesirable ideas. Non-release of poor quality R&D products 
should be a viable option for program administrators. Quality 
control is an important function of the model's product 
development stage. ^ • , 

An assessment of product distr ibu t ion can tell a decision 
maker if the right people are receiving the R&D resu^.ts. 
Sometimes people on the distribution list are not in a position 
to use the materials. An awareness leaflet describing the 
product may be more appropriate than sending them the entire 



product. O^ten, groups of professionals find uses for products 
not anticipated by develbpers. A program administrator should 
vi(^w product distribution lists as dynamic changing, entities. 

Product use should be evaluated periodically to determine if 
R&D results are being used by the intended audience. Circum- 
stances sometimes prevent full acceptance of materials, e.g., the 
teacher may not know how to present the information or funds may 
not be available to purchase supplies essential to the use of the 
product. R&D program managers should' conduct evaluations to 
determine if appropriate materials are being used to overcome 
high priority problems in vocational education. 

Assessment of product impact is more difficult than the 
other types of evaluations, but it promises to pay big dividends 
in several ways. First, an impact assessment can tell a decision 
maker whether practice in vocational education classrooms is any 
different as a result of using R&D products. A problem does not 
have to be completely resolved to have been influenced by the 
R&D. For example, a recent impact study of entrepreneurs.hip 
materials (Plull 1983 ) showed some progress being made in infusing 
entrepreneurship materials into classrooms. In one instance, a 
separate course on starting one's own business was becoming 
institutionalized as a result of using the R&D products being 
studied. These "readings" on state-of-the-art practice in the 
field serve not only as an indication of what has changed, but 
also as a baseline measure for planning the future. 
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The second difference that impact assessment can make in R&D 
programming is related to baseline measurement. Impact studies 
can provide a kind of needs assessment to determine whether more 
work ought to be done in a problem area. Tf the site visits 
conducted during the impact study of en trepreneurship education 
reveal positive effects on students, for example , a state 
department of education may want to consider mounting a statewide 
program in this area. Impact assessments can therefore help in 
establishing priorities for the future. 

Third, impact assessments can serve as a summative 
evaluation of dissemination/diffusion processes. User audiences 
may be missed during the distribution of R&D products, teachers 
may leave products on the shelf following an initial 
"bandw:agon-style" introduction of a product into a school, or 
resources may not have been available for full Implementation of 
the new idea. Such problems often surface during an impact 
assessment . Program improvement processes in vocational 
educa t ion can be upgraded if in vest iga tors are familiar with the 
process.es and are willing to analyze data with these processes in 
mind . 

• There are iuany obstacles to conducting successful impact 
studies. The studies are expensive, longitudinal investigations 
require advance planning, and results may reflect unfaybrably on 



persons responsible for program improvement, thereby creating 
potential problems for utilization of results. But the logic of 
assessment as a means of accountability in vocational educciLion 
R&D is compelling. Wise administrators at the local, state, or 
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national level will take steps to implement impact assessments in 
a manner- consistent with the time and cost constraints of their 
situation. 

Locals State^ and National Partnerships 
The successful implementation of a vocational education R&D 
evaluation system will require the cooperation of all levels of 
government. Local project directors will need to comply with 
requests for distribution data and, on occasion, development 
data. State research administrators must take care to outline 
expectations for impact in Requests for Proposals." 

State education departments represent a vi'ial link in the 
netv/ork of vocational education R&D. They not only sponsor 
research, but they must also interpret and use its results to 
advance education's interests. In addition, state agencies play 
an important role in providing accountability data to the federal 
government. Organizations such as the National Center and the 
regional curriculum centers provide expertise and technical 
assistance to states, the federal government, and to local 
projects on an "as needed" basis. Their activities should make 
the entire program improvement process in vocational education 
operate more effectively. 

V The question of who should conduct the product distribution, 
use, and impact assessments is an important question. Clearly, 
distribution data should be collected through routine channels 
and summarized with the least disruption possible. Knowing the 
number of copies distributed from R&D projects is important 
<• 
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information. The National Center and/or the U.S. Office of 
Vocational and Adult Education is in the best position to conduct 
and summarize these data. 

Utilization and impact studies differ greatly from 
dissemination data collectio^ requirements. Use and impact 
studies should be conducted as the need arises, but the funds 
always will be limited. It is not likely that the U.S. Office of 
Vocational and Adult Education will want to invest valuable staff 
time in conducting these studies. The National Center would have 
a comparative advantage for conducting these studies, because of 
the relationships it has established with state departments of 
education and with local education agencies. The National Center 
has also established an excellent track record in delivering 
high-qual ipty studies. The greatest amount of credibility would 



come from an independent third-party investigation of product 
impact on high priority questions. In any event, agreement on 
the area(s) to study must be reached in advance of funding, and 
that agreement must include standards for judging the performance 
of P.FiD projects and products. 

This report has addressed a number of significant questions 
associated with the conduct of vocational education RSiD impact 
studies. More questions have been raised than answered, but the 
possibility of improving on our current state of data collection 
seems likely. Systematic collection of distribution data, 
selected studies of product utilization, and investigations of 
product impact offer the promise of empirical obse vations to 
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justify or deny the tremendous dollar investments being made in 
the. vocational education R&D enterprise in this country. We 
should not hesitiate to move forward in this quest for an 
effective evalution system. 
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CHAPTFR 4 
SUMMARY 

This report contains a brief review of vocational education 
R&D legislative authorizations. Federal and state dollars are 
being invested in research and developnen t (including curriculuin 
deve lopme nt ) to improve voca t ional educ^; t ion programs , bu t 
evidence of impact has been scant. Individuals tend to adapt R&D 
products during use, and often the effects of use are embedded in 
organizational routines. So it is difficult to attribute causal 
changes in students and organizations to' the use of R&D products. 
VJhat is needed is an evaluation system that provides essential 
information on product distribution, use, and effects to relevant 
decision makers for planning and accountability purposes. 

The objective of this study was to design a simple and 

relatively low-cost evaluation system for judging the impact of 

R&n products and information. The system is to use available 

A' 

data whenever possible; take advantvige of existing reporting 
mechanisms; and assess the effectiveness of R&D at the local, 
Ktate, and federal levels. 

The methodology of this study relied on telephone interviews 
wi th product users, off-the-shelf records of product dis- 
tribution, and on-site interviews to gather data on R&D impact. 
These data are reported in a companion document entitled Summary 
of Dissemination Outcomes (Hul 1 1983 ) . Es t ima tes of RCU 
capabiTitTes~"arnd" those of other agencies in vocational education 
to evaluate R&n activities are based on these data. The next 
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sections present conclusions and recommendations regarding the 
readiness of government agencies to assume the data collection 
burden of a comprehensive and continuing evaluation system. 



Conclus ions 

The results of the study lead to the following conclusions 

o On -the -she If distribution data exist and can be 

accessed through relatively low-cost record keeping 
procedures. 

o Most RCU directors are willing and able to provide 
distribution data on the R&D products they 
distribute . 

o Most states are not organized to provide product 
distribution data by standard i zed categories, 

o Data on use of R&D products are more difficult to 
obtain than distribution data, 

o Superficial data on product us may be obtained by 
mail , but documenting the degree of product us»^ in 
detail usually requires interview and observation 
data • 

o Baseline measures, of some type must be established 
when assessing product impact. 

o Interactive methods (e .g . , telephone and face- to- face 
interviews) are necessary for collecting impact data 
under most conditions. 

o Data on product use and impact are collected by very 
few states at the present time. 

Recommendations 

Based on the conclusions of the study, the following 

recommendations are offered for establishing a vocational 

education R&D comprehensive evaluation system : 

o A comprehensive and continuing R&D product evaluation 

system should be implemented^ in ph_ases , beginning 
with the collection of distribution data. ^ . . 



St-.udies of product -use should he targeted to specific 
samples of product recipients. 

Impact studies of the effects of product use should 
be funded for high priority needs only. Enough money 
■should be available to the project for the collection 
of baseli ne da ta. 

Stud ies a imed at es tablish i ng general s tarda rds for 
evaluating dissemination iptrategies, utilization 
approaches, and effects of product use need to be 
conducted. General classes of standards for 
evaluating these activities can be used as a moBel 
for deriving specific standards for projects. 
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.. TABLE A-1 
SUMMARY OF. INSTRUMENT USE 



Type of Instrument 


Respondent 
Population 


Appropriate 
Number of 


Total Nun 
of Burden 
Hour s 


)evelopment Survey 


Product Developers 
and Sponsors 


* 


* 


)istribution Survey 


Product Developers 
and Dif.tributors 


50 


133 


Jse Survey 


Product Recipients 

. I 


15,000 


5,000 



* 

The development survey will be used to supplement the information from 
the distribution survey only when there is special interest in a 
particular product. It would be administered to one or two respondents 
per product and would take twenty minutes to complete. 
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Tear Sheet for Telephone InterAriews 



Product Development/Dissemination 
Interview Schedule 

Telephone Interview Date 

Respondent " • Prod^ict Id. 

Role Produce fyp 

Organization 

City /St ate 



R&D Product Development Survey 



The National Center for Research in Vocational Education at The Ohio State 
University is conducting a} study to determine the use of selected 
research and development products. This survey concerns the product 
Identified in the enclosed] description. Please complete the questions 
about this product by eith'er circling the appropriate response or 
filling in the blank space provided. Your voluntary participation is 
appreciated. 



1. Was a written needs assessment completed prior to the decision to develop 
= this- product? 

a Yes 

□ No 

2. Are objectives for product use clearly stated early in the product? 

□ Yes ^ 

□ No • 

3. Were primary target audienc'es identified prior to development of the product? 

□ Yes 

□ No"^^ 1 

I ■ . ■ 

If yes, who are they? I 

' :_ i 

/ ■ ^ 

4.. Were computerized databases searched (especially Research in Vocational 
Education) prior to product development? 

' • □ ^ Yes 

□ No 

'Was other literature reviewed? 

^ ■ \ 

□ Yes 

D No ' ^ 

If yes, list sources (e.g., Dissertation Abstracts) . 



Number of citations used in developing product 



Form Approved 
FEDAC No. 
App. Exp, 



Survey No, 
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5. Who developed the product? (Number of people by role and organization) 



Role 



Organization 


Role 


Administrator 


Teacher 


Student 


Other 


Education Organizations 

State education agencies 

Secondary schools, in- 
cluding districtwide 
audientes 

Two-year community/ 
technical institutes, 
proprietary schools, and 
programs for adults 

Universities and other 
four-year institutions 












Other Public Organizations 

Military, correctional in- 
stitutions, rehabilitation 
siervices, CETA/JTPA pro- 
grams, and community-based 
organizations such as in- 
dustry opportunity centers 












Other Private Sector 
Organizations 

Business, industry, and 
labor councils 


«r 











6. Was the product revised? 

□ Yes 

□ No 

How many revision cycles? 

How many people provided feedback for revisions'' 

7. What evidence do you have that the product is of high quality? 
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8. Are estimated costs of product implementation available? 

□ Yes 

□ No 

If yes, how did you arrive at them? 



9. riow would you rate the product on the following indicators of quo 

Very 
High 
Qualitj( 

5 



Unaccep- 
table 



Low 

Quality 



Scholarship 12 3 4 

Usefulness 12 3 4 5 

Equity 1 2 3 4 5 

Communi cabi 1 Ity 1 2 3 4 5 

Marketability 12 3 4 5 

Overall Quality 1 2 3 4 5 



y: - 

No 

Information 
9 
9 
9 
9 
9 
9 



10. How many copies (total) have been disseminated since the product was 
released? ^ Date of initial release 



11. Where did the copies go? 



Organization 


Rcle 


Role 


A(lministrator 


Teacher 


1 

Student 


Other 


Education Organizations 

State education agencies 

Secondary schools, in- 
cluding districtwide 
audiences 

Two-year community/ 
technical institutes, 
proprietary schools, and 
programs for adults 

Universities and other^ 
four-year institutions 












Other Public Organizations 

Military, correctional in- 
stitutions, rehabilitation 
services, CETA/JTPA pro- 
grams, and community-based 
organizations such as in- 
dustry opportunity centers 












Other Private Sector 
Organizations 

Business, industry, and 
labor councils 
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12. What were the primary means of product dissemination? How many copies^ 
were distributed by each means? 

Number of copies 

Q Direct mail 

□ Conferences and workshops ' 

p Resource centers 

□ Technical assistance ■ 

□ Commercial publishers " 

□ Development site 



TOTAL 

13. Did you encounter any unusual circumstances in the distribution of this 
product that may have enhanced or inhibited its use by recipients? 



Thank you for your helpl Please return this completed survey 
in the enclosed self-addressed stamped envelope. 



62 



69 



EKLC 



Form Approved 

FEDAC No. 

App. Exp. 



R&D PRODUCT DISTRIBUTION SURVEY 

Date this ^ecord was completed 

Product .itle 



Author 



Publication Date 



INSTRUCTIONS: Please complete this information and attach it to the product when_. 

the product is submitted for inclusi_on_ij3_±he--ERI-G"^-e ^ 
database. These records document the initial distribution of 
copies immendiately following publication release. Thank you^for 
voluntarily sharing this information. 



1 . Type of Report: 

1. p Research Report (Primary research including evaluations) 

2. □ Resource Guide/Directories/Bibliographies/Literature Review 

3. □ Policy or Position Paper 

4. Q HandbooK/Procedura"! Guide 

5. □ Teacher Guides 

6. Q Student Material 

2. Substantive Focus: 



01 Vocational Education Service 
Area 

02 Economic Developrujnt 

03 Technology 

04 Business/Industry Training 

05 Linkages 

06 Special Populations 

07 Curriculum and Instruction 

3. When was the product released? 



08 Equity 

09 Career Development 

10 Fundamental Skills 

M Adult Training/Education 

12 General Information 

13 C .her program imprc/ement processes, 
such as evaluation, follow-up, 
management, administration, and 
planning 



How many copies of this product were distributed during the remaining months 
of the calenaar year after the product was'reT&ase^ 



C3 



EKLC 



70 



5. To whom were these copies distributed? 



Role ^ — 

Organization 


To tal 


Number of Copies 


Administrative, 
Professional, and 
Support Personnel 


Teachers/ 
Trainers 


Students, 
Trainees 


Education Organizations 

State education agencies 










Secondary schools, including 
^___d±str4c^^ide--a:ud±elic e s 

Two-year community/technical 
institutes , proprietary 
schools, and programs for 
adults 

Universities and other four-year 
institutions 




- 






Other Public Organizations 

Military, correctional insti- 
tutions , rehabilitation 
services, CETA/JTPA programs, 
and community-based organi- 
zations such as industry 
opportunity centers 










Other Private Sector Organizations 

Business, industry, and labor 
councils 










Total 











6. In. addition to the copies indicated above, how many copies were sent to the 
following: 

Persons at the national or regional level 
Persons in other countries 



7. Which of the following methods were used to distribute the product? (Check 
al 1 that apply. ) 





1. 


Direct mail 


CZ3 


2. 


Workshops or meetings 


[=1] 


3. 


Personal contact 


CZl 


4. 


Development centers 


C=i 


5. 


Libraries, resource centers 


en 


6. 


Commercial publishers 
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Teor sheet for telephone interviews 

Telephone Survey Identification Survey No. 

(to be discarded following interview) 

Telephone Interview Date 

Recipient Id. Product Id. 

Role — ^I-rvt-ervi-evrer 



Organization 
City/State 
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Form Approved 

FEDAC No, 

App. 



Exp, 



R&D PRODUCT USE SURVEY 



Survey No. 



The National Center for Research in Vocational Education at The Ohio State 

University is conducting a study to d etermine th ^jJse_p_fL selected — 

xesj&ari:J:5_arKl-^eve1apment-TJ^ This survey^ concerns the product 

identified in the enclosed description. Please complete the questions 
about this product by either circling the appropriate response or 
filling in the blank space provided. Your voluntary participation is 
appreciated. 



Do you remember the product? p Yes □ No (If no, answer no further questions 

and return the forms.) 

Date the product was received 



Were you involved in the product's development (write, test, review, etc.)? 
p Yes □ No 



How did you learn about the product? (Check all that apply) 
□ Brochure, flyer 



□ Displays at conventions 

□ Personal contact 

□ Technical assistance 



p Announcements » articles in 
newsletters or periodicals 

□ Workshops, conferences 

p Ccrntact with a development site 

p Through an information network 



How did you receive this product? 

p My supervisor gave it to me. 

p It was in the file. 

□ It was sent unsolicited in the mail. 

p I ordered it. (What motivated you to order this product?) 



Have you used any of the ideas contained in the product? 
If not, skip to question 10- 

□ Yes ' 

□ No 
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7. How have you used the product? (Check 

□ 1 Read, reviewed, or studied it 

□ 3 Improved instruction or other 

curricular/counsel ing activities 

p 5 Influenced planning or policy 
deci sions 

□ 7 Placed in a library or resource 

center 

□ 9 No use. (Why was the product not 



all that apply .) 

□ 2 Referenced or quoted from it 

□ 4 Planned or conduc*:ed researjh 

□ 6 Effec-ed changes in the 

organization 

□ 8 Shared the product for other's 

use 

ised?) 



8. If you 'ihared the product or the information in it with others, who were 
they, and what type of use took place? 



Write the number of users annually in each cell 



(Check the type of use.) 


a 

Administrator 


b 

Teacher 


c 

Studert 


d 

Other 


□ 1 Used in an instructional 
setting (class* workshop* 
etc.) How many hours of in- 
struction annually related 
directly to this product? 










□ 2 Used in a noninstruc- 
tional way to improve 
programs or the organi- 
zation. 










□ 3 Shared informally with 
colleges for the'^r own 
personal use. 











a. Include superintendents, principal:, coordinators, supervisors, etc. 



b. Include instructors, professors, sttifnt teachers, etc. 

c. Inclurip ^'1 learners. ( e.g. , teachers in ^n in-ser\ice workshop). 

d. Include parents, board members, advisory groups, counselors, etc. 

. What plans do you have for future use of the product' 



SLop.. l.'iis completes the questionnaire if use has not taken place. 
P'ease return Volt coTnments in the self-addressed stamped envelope. 
P ise has taken ,vace, go on to the next question. 
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10. If you used the product, was it modified in soue 
o Yes 

□ No ^ 

11. What was the degree of modification? 

Very Little 

0^ 

•12; How many copies of the product were repriu^,ed 7 Ordered ? 

13. Did you receive any support within or ou:s*idfc of the organization to help 
use the product? 

□ Yes 

□ No 

(If yes, check all that apply:) 

□ 1 Training j-, 2 Technic^' assistance 

□ 3 Administrative support □ 4 Financial support 

□ 5 Release time p o Access 10 needed supplies, equipment, 

ari spaci 

U 7 Other 

14. Did you need support? 

□ Yes 

□ No 

15. Was the level of support adeqta c? 
n Yes 

n No , 

E .timate the annual costs to you or yoi r organization to use this product. 
(Flespond to all that apply.) 

1 Purchase price of the pro^'uct ($ ) 

:j 2 Teacher preparation time (estimated number of hours ) 

□ 3 Othor training costs ($ ) 

□ 4 Annual costs for supplies, equipment, space ($ ) 



Very Great 

-a^ ■ 
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17. How would you rate the product on the following criteria? 



... ■. « 


Poor 


Fair 


Good 


Very 
Good 


Excellent 


1. Relevance (meets an implicit 
or explicit need) 




2 


3 


4 


5 


2. Utility (can be used by you' 
or others ) 


1 


2 


3 


4 


5 


3- Contribution to knowledoe 
(in the profession or to 
you personally) 


1 


2 


3 


4 


5 


4. Scholarship (accurate and 
reliable content) 


1 


2 


3 


4 


5 


5. Value in. relation to cost 


1 


2 


3 


4 


5 


6. Editorial quality 


1 


2 


3 


4 


5 


Compared to similar products, 

rate this product on overall quality. 


1 


2 


3 


4 


5 



19. How could this product or its delivery to you have been improved? 



20. Describe changes resulting from the use of this product in the following areas: 
1. Your personal knowledge or skills 



2. Use of time 



3. Student learning 



4. Organizational efficiency 



5. Equity for special populations (e.g., the handicapped, minorities, women) 



6. Business and industry 



A MORE COMPLETE DESCRIPTION CAN BE WRITTEN ON THE REVERSE SIDE OF THIS PAPER. 
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Estimate the annual amounts of money or time saved as a result of using this 
product. 

$ . hours. 



What is the basis /for these estimates? 



What are the chances that use of this product can improve vocational education 
over time? 

Not Very Don't 
Likely Likely " Know 
□ 1 qS ^4 p9 



Thank you for your helpl Please return this completed survey 
in the enclosed self-addressed stamped envelope. 
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A CONCEPTUAL FRAMEWORK FOR ASSESSING IMPACT 

*} 

There are many diverse organizations in vocational education 
that produce research and development products for use in the 
ti^l6. These include curriculum centers in. state departments, the 
National Center, and state universities, among .others. Figure • 
B-1 displays r.il a t ionsh i j)S among these organizations in the 
production, distribution, use,, and dmpact of R&D products. 
Federal funds flow 'from thelj^S.jCotfice of Vocational and Adult 
Education to states for program improvement. Often these federal 
funds are matched with -state or local dpllars. The types of 
projects actually conducted in these fifty different funding 
points vary greatly, depending on the amount of money available 
and the research needs of that particular state. ' 

An average of $380, 502 was allocated for research, demon--, 
stration of exemplary pro'ducts., and curriculum development by 
each state per year (FY 1978-1982), according to Budke (1983). A 
groat deal of R&D variance is noted among the states, with larger 
states averaging 52,637,507 and smaller states averaging $48,589 
for a comparable., time period. 

Vocational education R&D results in diverse types of 
products, as exemplified by the following: 

o Research rind evaluation reports 

o Review and synthesis cjf research, including 
secondar 7 analysis 

o Policy papers 

o Position papers 
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U.S. 

C ice of 
Vc Ciitional 
and Adult 
Education 



Regional 
CCCs 



NCRVE 



State 
Education 
Departments* 



Post secondary I 
Institutions 




— 1 


/ 

/ J 




/ / 
' — ^ / / 



State 
Jniversities*! 



LEAS 



Figure B-1. , Interface relationships among institutions 
concerned with quality and utility of 
vocational education R&D. 

* Indicates major producers of R&D products 
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o Resource guides, directories, and biblio- 
graphies 

o Handbooks and procedural guides 

o Teaching outlines and training packages 

o Student learning materials 

Expectations for product impact depend on the substantive 
nature of the document, the reasons for its use, and the 
particular setting in which it is used. This diversity of actors 
and products raises concerns about technical problems in 
aggregating evaluative data across highly variable conditions. 
As in the Vocational Education Stud y (1981)r variability among 
procedures used to secure placement data raises questions about 
the validity and comparability of local and state aggregations. 
What is needed is an R&D product evaluation system that focuses 
on essential information for planning and accountability 
purposes. The system should be relatively simple and low-cost, 
using existing organizations and relationships whenever possible. 

As an aid in organi-zing- ideas about impact, a conceptual 
framework is depicted in figure B-2. It contains salient 
features of impact producing processes. These features are 
grounded in change process literature, and particular attention 
is focused on change as a result of using new ideas from R&D 
projects. This conceptualization assumes that (1) new ideas can 
be packaqed in a transportable format for use in diverse settings 
and (2) R&D products can be used by persons not involved in their 
development. Cost reduction efficiencies are envisioned by 
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Figure B-2. Conceptual framework 
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sharing high-quality products with others, thereby avoiding 
"rediscovery of the wheel," 

The conceptual framework specifies impact-producing 
processes. Activities and events are believed to be cumulative 
in effecting change based on the use of R&D products. The 
framework in figure B^2 highlights, critical processes associated 
with impact. The processes are somewhat sequential in nature 
(e.g., implemen tatfbn of a product following distribution), but 
there is also a sense of recycling within each process. For 
example, secondary di'^tr ibut ion of a product may occur when 
recipients share it with other individuals. Effectinc; change in 
vocational eduG^ation using R&D .products may also be influenced by 
unforeseen factors. For example, a budget crisis may intervene 
to disrupt the smooth implementation of an R&D product in a 
sch^^^ setting. What 'follows is a brief discussion of concepts 
in the framework. The remarks are based on information from the 
literature and experience in collecting impact data. 

Paramount among these concepts is the development of a high 
quality product . A three-year study (Louis , Rosenblum, and 
Molitor 1981) of the National Institute of Education's research 
and development utilization program found product quality to be 
particularly important in predicting the degree of problem 
resolution. Product quality is considered a measure of the . 
relevance of the product to the situation and the degree to which 
it is a genuinely new way^of doing things. These measures, taken 
at fifty-One sites during 1978 to 1980', used focused unstruc- 
tured interviews and included approximately 100 products or sets 
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of materials. The National Center's Advisory Council reviewed 
criteria in the evaluation model proposed in this paper and rated 
the development cr i ter ia at the very top of. the list. 

Qual i ty control of R&D products is difficult to achieve, 
however, and even more difficult to have accepted in the field as 
an important indicator of potential impact. As Klein (1978) 
points out : 

. . .users are not systematic in their approach, and 
seldom use effective ness criteria. My experience 
• working with user groups , such as teachers , supports 

the notion that the users feel very ungual if ied to 
look at even summary reports on evaluation. (P. 119) 

Quality control of R&D products is an uphill battle for 
developers. Money is scarce ,. and _some people assume that money 
allocated for product production automatically results in a good 
product. What is needed is more evaluat ions of products while 
they are being developed, with clear indicators of what the 
products can do for potential users. The great need for quality 
standards is documented in the Committee on Vocational Education 
Research and Development (1976) report and has. been emphasized 
more recently by Worthingtoh (1981). A comprehensive evaluation 
system must give attention to the development of standards for 
quality control. 

Widespread distribution of R&D products is implicit in the 
concept of comprehensive dissemination,, as defined by the 
Dissemination Analysis Group (DAG) (1976). DAG's four-level 
definition of dissemination (i.e^, spread, exchange*, choice, and 
implementation) has been endorsed by professionals in the field 
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of educational dissemination at the June 197.7 Dissemination 
Forum. 

The DAG conceptual definition of comprehensive dissemination 
forms an underlying rationale for comprehensive impact, as 
captured in the conceptual framework in figure B-2, The 
fiLSt-level definition of spread resembles a one-way casting out 
of knowledge similar to the idea of widespread dissemination used 
in the impact concep-tual framework. Impact does not take place 
in a vacuum. Antecedent conditions and concurrent factors, such 
as opportunity costs of resources used, influence the likelihood 
of impact from R&D products. What are some of these conditions? 
What are the requirements for effective program improvement as a 
result of R&D product use? Answers to these questions will be 
uncovered as more and more impact studies are conducted. 

Another concept relating to impact from R&D products is 
effective implemen ta t ion of the product in diverse educational 
settings. Some people subscribe to the view that good products 
will sell themselves. But product developers do not always know 
how products will be used. As articulated in the DAG (1976) 
report, products are selected by users for program improvement 
interventions. R&D products must be accompanied by support 
systems that involve physical and financial resource allocations. 
Product implementation and use must be endorsed by those in 
authority for institutionalization to occur. 

Fullan and Pomfret (1977), in their review of curriculum 
implementation studies, identify several factors influencing 
effective implementation. 
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Among them are the following: 

o Characteristics of the innovation, such as its 

complexity or difficulty of change required by the 
innovation 

o Strategies of implementation deal'ng with resource 
support, timing of the interventior, ^ feedback 
mechanisms, and participation in the innovation 
process 

o Characteristics of the adopting units, such as their 
demographics and ability to solve problems 

o Macro-sociopolitical factors, such as incentive 
systems , the role of evaluation , and pol i t ical 
compl ex i ty 

In an insightful article on incentives for innovation in the 

public schools, Pincus ( 1974 ) review^i bureaucratic f'xctors 

supporting innovation and characteristics of institutional 

settings. He concludes by saying: 

. . .in a diverse society. . .at any one time there will 
be a variety of standards. A major focus of R&D policy 
should be. .experime.itation and. . .incentives that 
encourage new patterns of institutional behavior, (p. 139) 

The possibility of an RSfD product being modified in the 

process of becoming adopted by an institution is always very 

great. The Rand Corporation (Herman and McLaughlin 1978) 

conducted a study that included the Part D Exemplary Programs 

authorized by the Vocational Education Amendments of 1968. The 

study showed effective projects to be characterized by mutual 

adaptation of both the product and the adoption site during the 

implementation process. The study also found professionalism to 

be a primary motivation when teachers undertake extra work. 

Clarity of objectives 1 ike wise had a major ef f ect on imple- 

mentation. In addition, comprehensive projects were found 




to be no more effectively implemented than simple pro jects ; in 

/ 

fact they wert^ somewhat less likely to. produce teacher change, 
The«e findings suggest the value of in -renriental goal -.>.^tting when 
introducing an R&D produce into- an established educat*. h1 
setting. 

Another concept re la ted to impact f rom R&D products i s 
sustained use of the R&D product . Caus . ... •.;Odels cons true te-.' ' 
Crandall et al. ( 1982) to explain finding- f om a sample oW ' -U' 
schools in ten states show that teacher cOi, ^ v-.r^ent and elapsr.^ -.i 
time (i.e., the length of time the teacher r-^^^ been using t^e 
innovation) are significant predictors of chanae in practice. 
The central ity and Importance of teacher commitraent or ownership 
are underscored by consistent patterns in subsets of the data. 
Organizational change, on the other hand, reflects the importance 
of the principal's management style and leadership. The 
principal's ability to adopt a *'tr':<e charge" attitude provides 
the only route to institutionalization of R&D-based innovations. 

What figure B-2 does .lot slow Is the time frame necessary 
for changes to be brought 'ihout bj* the use of R&D products. 
Figure B-3 places the fac^tors affecting i \pact in a linear 
sequence. Types of eva luacioPK act ivi ties (are ind\cated. 




including needs assessment, which h^^^ejari added as a basis for 

judging the effectiveness of the impact. Two important '"jon- 

siderations in the e^valuative process are indicated in ,.he figure 

as decision processes. They are (1) the decision to release a 

product and (2) elapsed time between introduction of a pro Juct to 

a setting and ev/aluation of that setting for product impact, 
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I. DETERMINE PRIORITIES. IDENTIFY PRIMr lY USER AUDIENCES. 
AND ESTABLISH STANDARDS FOR EXCELLENCE (NEEDS 
ASSESSMENT) 



4* 



II. SEARCH FOR OR 
DEVELOP HIGH- 
OUALfTY 
PRODUCTS 



(QUALITY CONTROL) 



DOCUMENT 
DISSEMINATION 
OF RESULTS 



IV. ANALYZE 

IMPLEMENTATION 
AND USE OF 
PRODUCTS 



(FORMATIVE EVALUATION) 



t 



V. ASSESS THE 

IMPACT or 

SELECTED 
PRODUCTS 

(SUMMATIVE 

EVALUATION) 
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Figure B-3 . Linear Sequence for Evaluation Activities 




Figure B-3. Dimensions of Impact 
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Assessment of impact is difficult because effects of product 

use interact with other variables. Figur-"* B-4 illjrtrates 

several dimensions of impact. ,The two p-imary ones^ time and 

amount of change, have been recognized in the change process 

literature for at least a decade. They are included in the 

impact assessment design by Evans et al. (1982) and are noted in 

an early review of strategies for effecting change by Wall 

(1972). These two dimensions interact to explain much of the 

difficulty in measuring impact. 

Large amounts of impact are observable in the short run. 

The difficulty is that most R&D products produce relatively small 

impacts that tend to become embedded in a web of other variables. 

Student achievemetit scores are affected by aptitude, and the 

efficiency of a school system is determined in part by its size. 

Searching for impact data and attributing causal inferences to an 

R&D product demand careful attention to the relevance of 

variables and • the credibility of data sources. 

There are many constraints on the collection of impact data, 

including some of the following: 

o Impact is often embedded in organ i r ^t iona 1 
routines. 

■f 

o Time is required for impact to take place. 

o Data collection is expensive . ^ 

o Interviewers may need, to be trained. 
Constraints such as these o'^ten prohibit impact assessment. That 
is why data on product distribution and use should be examined as 
preliminary measures of i'^.-act. 
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\ GLOSSARY OF TERMS 

\ 

Continuing and comprehens i ve > Th is evalua t ion sys.tem should 
become operational for the forseeable future, collecting 
evaluative data necessary for accountability and planning 
purpoj^es. The operation of this system should be within the time 
and cost constraints of the federal government. 

Distribution . This one-way transfer of knowledge is* analogous to 
the spread of information and products defined as level one of 
dissemination by the Disseminat ion Analys is . Group ( 1976).. 

Impact' . Impact is change resul t ing - f rom use of products or 
information. It may be positive or negative,, short-term or 
long-term, anticipated or unanticipated. This effect of product 
use may affect individuals, organizations, or societies (Hull, 
Adams, and Bragg 1983). 

Information and products . There are outputs from vocational 
education R&D in the form of products; such as (1) research 
reports, (2) knowledge synthesis papers (analyses of research 
findings), (3) resource guides (cites/describes available 
materials), (4) admin is t rat ive/ implemen tat ion guides (manuals and 
handbooks for administrators), (5) instructional/implementation 
guides (manuals and handbooks for teachers), ^ (6) learner 
materials (instructional resources for students), and 
(7) conference proceedings (collection o^f presentations, 
speeches ) . • 

Innova t ion . Any activity perceived as new by an adopting unit.' 

National, regional, and state significance . For the purposes of 
this task, significance refers to products that have been 
submitted and accepted for inclusion in one of the three 
databases: Educational Resources Information Center (ERIC), 
Resources in Vocational Education (RIVE), or VocatiohaX Education 
Curriculum Materials (VECM). * . 

Programs of national significance . This . broad-based term 
includes nationwide projects, the curriculum coordination' 
centers, the National Center for Research in Vocational . . 
Education, graduate ^Jeadership development fellowships., a teacher 
certification program, and a National Occupational Information 
Coordination Committee. ^ . . ' 

R&D product . Any output from a project financed with 'federal 
program improvement vocational education funds. These funds are 
narrowly defined as Sections 131, 132, and 133 (and their 
counterparts in Section 171) in P.L. 94-482. 
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utilization . . Utilization is the integration of knowledge into 
personal thoughts and actions, organizational routines, and 
societal values. It encompasses the next three levels of the DAG 
definition of dissemination: exchange, choice, and imple- 
mentation. In fact, it goes beyond these concepts to include 
modifications in products and environments to upgrade product 
ef fe^c tiveness (Hall and Loucks 1977 ). 

Vocational education R&D . Prog ram improvement contract 
authorized by Subpart 3 of P.L. 94-482, Sections 131, 132, and - . 
133, and corresponding coritracts involving appl ied - research , 
curriculum development, and exemplary aind innovative programs, 
Subpart 2 of P.L. 94-482. Specifically excluded from 'this 
de f in i t ion are supportive services grants an.d contracts, such as 
vocational education personnel development and vocational 
guidance and counseling. 
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